Differentially Private Feature Selection via Stability Arguments, and the Robustness of the Lasso

We design differentially private algorithms for statistical model selection. Given a data set and alarge, discrete collection of “models”, each of which is a family of probability distributions, the goal isto determine the model that best “fits” the data. This is a basic problem in many areas of statistics andmachine learning.We consider settings in which there is a well-defined answer, in the following sense: Suppose thatthere is a nonprivate model selection procedure f , which is the reference to which we compare ourperformance. Our differentially private algorithms output the correct value f(D) whenever f is stableon the input data set D. We work with two notions, perturbation stability and subsampling stability.We give two classes of results: generic ones, that apply to any function with discrete output set; andspecific algorithms for the problem of sparse linear regression. The algorithms we describe are efficientand in some cases match the optimal nonprivate asymptotic sample complexity.Our algorithms for sparse linear regression require analyzing the stability properties of the popularLASSO estimator. We give sufficient conditions for the LASSO estimator to be robust to small changesin the data set, and show that these conditions hold with high probability under essentially the samestochastic assumptions that are used in the literature to analyze convergence of the LASSO.